Image processing device, image processing method, and intraocular image processing system

ABSTRACT

To provide an image processing device capable of observing the inside of the eye in a wider range and simultaneously. There is provided an image processing device including an integration processing unit configured to integrate ophthalmologic images acquired by a plurality of types of ophthalmologic image capturing devices by correlating intraocular positions and to generate one integrated image indicating intraocular information in a range wider than a range indicated by each of the ophthalmologic images.

TECHNICAL FIELD

The present disclosure relates to an image processing device, an imageprocessing method, and an intraocular image processing system.

BACKGROUND ART

In retinal surgical operations, while there is a demand for alwaysobserving a wide range simultaneously in order to determine retinaltraction by the vitreous body and the like, for example, there is ademand for enlarging and observing the posterior pole when providingtreatment for the posterior pole. For example, for posterior poleobservation of observing the posterior pole of an eyeball, a contactlens to be mounted on the eyeball, a non-contact lens, or the like isused. Meanwhile, for wide-angle observation of observing the fundus ofthe eyeball in a wide range, a wide-angle observation lens and the likeis used. The wide-angle observation lens is provided in a surgicalmicroscope as an additional optical system, for example, in a wide-angleobservation system for observing a real image created by the additionaloptical system. The observation range for wide-angle observation iswider than the observation range for posterior pole observation.

CITATION LIST Patent Document Patent Document 1: WO 2017/169283 SUMMARYOF THE INVENTION Problems to be Solved by the Invention

The use of the surgical microscope and the wide-angle observation systemmakes it possible to observe the inside of the eye in a wide rangesimultaneously, but the range of the observation is limited. Therefore,in a case where it is desired to observe the inside of the eye in awider range, in addition to images acquired by the surgical microscope,it is also considered, for example, to present an image acquired by animage generating device including an intraocular endoscope and the like.At this time, by showing which position in the eye each image indicates,it is possible to easily determine the position and direction of asurgical operation member. For example, Patent Document 1 discloses atechnique for displaying an image representing a relative posture of anintraocular endoscope in the eye, and an intraocular endoscopic imagecaptured by the intraocular endoscope.

With the technique described in Patent Document 1, it is possible todetermine which part of the eye is captured to create the intraocularendoscopic image. However, the technique of Patent Document 1 presentsan image representing the posture of the intraocular endoscope and theintraocular endoscopic image as separate images or in superimposition,and does not present a wider range of intraocular image than theintraocular endoscopic image.

Therefore, the present disclosure proposes a novel and improved imageprocessing device, an image processing method, and an intraocular imageprocessing system that enable observation of the inside of the eye in awider range and simultaneously.

Solutions to Problems

According to the present disclosure, there is provided an imageprocessing device including an integration processing unit configured tointegrate ophthalmologic images acquired by a plurality of types ofophthalmologic observation devices by correlating intraocular positionsand to generate one integrated image indicating intraocular informationin a range wider than a range indicated by each of the ophthalmologicimages.

Furthermore, according to the present disclosure, there is provided animage processing method including: integrating ophthalmologic imagesacquired by a plurality of types of ophthalmologic observation devicesby correlating intraocular positions; and generating one integratedimage indicating intraocular information in a range wider than a rangeindicated by each of the ophthalmologic images.

Moreover, according to the present disclosure, there is provided anintraocular image processing system including: a plurality ofophthalmologic observation devices configured to acquire ophthalmologicimages during an ophthalmologic surgical operation; and an imageprocessing device configured to integrate a plurality of types of theophthalmologic images acquired by the ophthalmologic observation devicesby correlating intraocular positions and to generate one integratedimage indicating intraocular information in a range wider than a rangeindicated by each of the ophthalmologic images.

Effects of the Invention

As described above, the present disclosure enables observation of theinside of the eye in a wider range and simultaneously. Note that aboveeffects are not necessarily restrictive, and in addition to or insteadof the effects described above, any of the effects indicated in thepresent specification or other effects that can be determined from thepresent specification may be produced.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an explanatory diagram showing an example of intraocularobservation lenses to be used in posterior pole observation andwide-angle observation.

FIG. 2 is an explanatory diagram showing an observation range of theposterior pole observation and an observation range of the wide-angleobservation with respect to a fundus range represented by flattening afundus part of an eyeball.

FIG. 3 is an explanatory diagram showing an example of use ofintraocular observation by an intraocular endoscope.

FIG. 4 is an explanatory diagram showing a display example of a positionand direction of the intraocular endoscope of FIG. 3.

FIG. 5 is a block diagram showing one configuration example of anintraocular image processing system according to one embodiment of thepresent disclosure.

FIG. 6 is a flowchart showing a flow of image integration processing andpresentation image generation processing by the intraocular imageprocessing system according to the embodiment.

FIG. 7 is an explanatory diagram showing generation processing of athree-dimensional integrated image.

FIG. 8 is an explanatory diagram showing one example of atwo-dimensional integrated image.

FIG. 9 is an explanatory diagram showing a right eye image and a lefteye image captured by an image capturing device mounted on a surgicalmicroscope.

FIG. 10 is an explanatory diagram for describing an image feature forestimating a posture of an eye in a surgical microscope image.

FIG. 11 is an explanatory diagram for describing a depth range to beused for generating a front image in a tomogram acquired by atomographic acquisition device.

FIG. 12 is an explanatory diagram showing one example of the front imagegenerated from the tomogram acquired by the tomographic acquisitiondevice.

FIG. 13 is an explanatory diagram showing one example of a vascularplexus image generated from the tomogram acquired by the tomographicacquisition device.

FIG. 14 is an explanatory diagram showing retinal depth informationobtained from the tomogram acquired by the tomographic acquisitiondevice.

FIG. 15 is an explanatory diagram showing an example of a position of aspecified region of the eye obtained from the tomogram acquired by thetomographic acquisition device.

FIG. 16 is a schematic explanatory diagram showing the intraocularendoscope in which a three-dimensional marker is attached to a gripportion.

FIG. 17 is an explanatory diagram showing one example of an endoscopicimage, and a range and direction of the endoscopic image with respect tothe fundus range identified from an estimation result of a posture ofthe intraocular endoscope when the endoscopic image is acquired.

FIG. 18 is an explanatory diagram showing one example of an illustrationshowing the fundus.

FIG. 19 is an explanatory diagram showing one example of a projectionmethod from a solid image.

FIG. 20 is an explanatory diagram showing another example of theprojection method from the solid image.

FIG. 21 is an explanatory diagram showing another example of theprojection method from the solid image.

FIG. 22 is an explanatory diagram showing one example of geometrictransformation of an integrated image.

FIG. 23 is an explanatory diagram showing another example of thegeometric transformation of the integrated image.

FIG. 24 is an explanatory diagram showing another example of thegeometric transformation of the integrated image.

FIG. 25 is an explanatory diagram showing one example of image workbased on a type of image.

FIG. 26 is an explanatory diagram showing one example of the integratedimage and the endoscopic image arranged side by side.

FIG. 27 is an explanatory diagram showing another example of theintegrated image and the endoscopic image arranged side by side.

FIG. 28 is a hardware configuration diagram showing a hardwareconfiguration of an image processing device according to the embodiment.

MODE FOR CARRYING OUT THE INVENTION

A preferred embodiment of the present disclosure will be described indetail below with reference to the accompanying drawings. Note that inthe present specification and the drawings, components havingsubstantially the same functional configuration are denoted with thesame reference symbol, and redundant description thereof will beomitted.

Note that the description will be made in the following order.

1. Observation range in intraocular observation

2. System configuration

3. Image processing

-   -   3.1. Image integration processing    -   (1) Image acquisition        -   Image acquisition by surgical microscope        -   Image acquisition by tomographic acquisition device        -   Image acquisition by intraocular endoscope        -   Image acquisition by illustration generating device    -   (2) Generation of integrated image    -   3.2. Presentation image generation processing        -   Flattening solid integrated image        -   Geometric transformation processing        -   Image work indicating type of original image        -   Integrated image and single image arranged side by side

4. Hardware configuration

5. Conclusion

[1. Observation Range in Intraocular Observation]

In retinal surgical operations, while there is a demand for alwaysobserving a wide range simultaneously in order to determine retinaltraction by the vitreous body and the like, for example, there is ademand for enlarging and observing the posterior pole when providingtreatment for the posterior pole. For example, for posterior poleobservation of observing the posterior pole of an eyeball E, a contactlens 10 a shown on the left side of FIG. 1 or a non-contact lens (notshown) is used. Meanwhile, for wide-angle observation of observing afundus of the eyeball E in a wide range, a wide-angle observation lens10 b and the like shown on the right side of FIG. 1 is used. Thewide-angle observation lens 10 b is provided in a surgical microscope asan additional optical system, for example, in a wide-angle observationsystem for observing a real image created by the additional opticalsystem.

FIG. 2 shows an observation range Aa of posterior pole observation andan observation range Ab of wide-angle observation with respect to afundus range Ao represented by flattening a fundus part of an eyeball.As shown in FIG. 2, the observation range Ab in wide-angle observationusing the wide-angle observation lens 10 b is wider than the observationrange Aa in posterior pole observation using the contact lens 10 a. Notethat for the sake of convenience, FIG. 2 shows only the optic disc andthe ora serrata as features of the retina, and this is similar in FIGS.4 and 17 as described later.

As shown on the right side of FIG. 1, the use of the surgical microscopeand the wide-angle observation system makes it possible tosimultaneously observe the inside of the eye in a wide range to someextent. However, even if the surgical microscope and the wide-angleobservation system are used, it is difficult to simultaneously observe awider range including, for example, an ora serrata 58, which is aboundary between a photoreception portion of a retina 57 (pars opticaretinae) and a non-photoreception portion (pars caeca retinae or parsciliaris retinae), or a pars plana 59.

Therefore, it is also considered to observe a portion that is difficultto observe in a case where the surgical microscope and the wide-angleobservation system are used from an image acquired by an imagegenerating device including an intraocular endoscope, an opticalcoherence tomography (OCT), or the like. For example, if the intraocularendoscope is used, it is possible to observe the pars plana 59 and thelike. If the OCT is used, for example, even in a case where the retina57 cannot be observed with the surgical microscope because see-throughis difficult with visible light, a retinal image can be acquired in somecases. However, in an image obtained with the intraocular endoscope, arange in which simultaneous observation is possible is extremelylimited. Furthermore, in a case where the OCT is used, it is currentlydifficult to obtain an observation range equivalent to an observationrange of the surgical microscope.

Furthermore, in order to switch between the posterior pole observationand the wide-angle observation as shown in FIG. 1, the lens to use needsto be changed, and the observation range in the posterior poleobservation and the observation range in the wide-angle observationcannot be observed simultaneously. It also takes time to change thelens.

Moreover, in a case where the wide-angle observation lens 10 b as in theright side of FIG. 1 is used, in an intraocular image presented to anoperator, the retina, which is originally close to a spherical surface,looks planar. Therefore, in a case where an unskilled operator looks atthe intraocular image, there is a risk of a wrong operation in which theoperator who recognizes that the retina is a plane moves a surgical tooland hits the surgical tool against the retina. Furthermore,characteristics of image distortion in the image observed by theoperator are determined by the optical system used during theobservation, especially in the surgical microscope, which is notnecessarily desirable for a user.

Furthermore, when the intraocular endoscope is used during observationof an intermediate translucent body opaque eye or during observation ofa peripheral portion, the range of image to be acquired is determinedfrom a tip position and direction of an endoscope probe with respect tothe eyeball, and from a posture of the endoscope probe represented by arotation angle around an optical axis. However, it is difficult todetermine the posture of the endoscope probe from the image captured bythe intraocular endoscope. Therefore, using the intraocular endoscoperequires a high degree of skill. For example, as shown in FIG. 3, it isassumed that an intraocular endoscope 20 is inserted into a vitreousbody 55 of the eyeball E from the right side of sheet of FIG. 3 at apredetermined rotation angle such that a tip 23 extending from a gripportion 21 of the intraocular endoscope 20 is located near a crystallinelens 53. At this time, even if a range B is captured by the intraocularendoscope 20 such that a direction of an arrow is upward in the image asshown in FIG. 4, it is difficult to understand this from the capturedimage.

Therefore, in view of the above, an intraocular image processing systemof the present disclosure enables intraocular observation in a widerrange and at the same time in a desirable way for the operator.

[2. System Configuration]

To begin with, with reference to FIG. 5, a configuration of anintraocular image processing system 1 according to one embodiment of thepresent disclosure will be described. FIG. 5 is a block diagram showingone configuration example of the intraocular image processing system 1according to the present embodiment. As shown in FIG. 5, the intraocularimage processing system 1 according to the present embodiment includesan image processing device 100, a plurality of image generating devices(hereinafter, the plurality of image generating devices will becollectively described as “image generating devices 200”), an inputdevice 300, and an output device 400.

(Image Processing Device)

The image processing device 100 includes an interface unit 110, acontrol unit 120, an integration processing unit 130, and a presentationprocessing unit 140.

The interface unit 110 is a functional unit that exchanges informationwith a user of the intraocular image processing system 1. For example,the interface unit 110 outputs an instruction to the system input by theuser via the input device 300 to the control unit 120. Furthermore, forexample, the interface unit 110 outputs a state of the system to theoutput device 400 to notify the user.

The control unit 120 controls the image processing device 100 on thebasis of the instruction from the user, and executes processing such asimage integration or generation of a presentation image in anappropriate mode. In addition to the image processing device 100, thecontrol unit 120 may control other components of the intraocular imageprocessing system 1 including the image generating devices 200 and thelike. For example, the control unit 120 may perform control such thatimage acquisition by the image generating devices 200 is appropriatelyperformed.

The integration processing unit 130 integrates images generated by theimage generating devices 200 by correlating intraocular positions, andgenerates one integrated image indicating intraocular information in arange wider than the image range of each image. The integrationprocessing unit 130 may perform alignment between images, for example,on the basis of image features included in the images generated by theimage generating devices 200. Furthermore, when integrating the imagesgenerated by the image generating devices 200, in addition toinformation obtained from these images, with reference to controlinformation of the image generating devices 200 and the like, theintegration processing unit 130 may estimate a posture of the eye or theendoscope, or may estimate a solid shape of the eye. The integrationprocessing unit 130 outputs the generated integrated image to thepresentation processing unit 140.

The presentation processing unit 140 processes the integrated imageaccording to a presentation format of the integrated image to generate apresentation image. The presentation processing unit 140 performs imagework including projective transformation, geometric transformation,color transformation, and the like on the integrated image on the basisof an instruction from the control unit 120. The presentation processingunit 140 outputs the generated presentation image to the output device400 via the interface unit 110.

(Image Generating Device)

The image generating devices 200 are devices that generate images to beintegrated by the image processing device 100 in the intraocular imageprocessing system 1. The image generating devices 200 include at least adevice that can generate an ophthalmologic image that is anintraoperatively acquired image acquired during an intraocularophthalmologic surgical operation (so-called modality). The imagegenerating devices 200 are specifically ophthalmologic observationdevices including a surgical microscope, a tomographic acquisitiondevice, an intraocular endoscope, or the like. Furthermore, the imagegenerating devices 200 may each include an illustration generatingdevice and the like that generates an illustration in which the fundusis illustrated. The image generating devices 200 are, for example,controlled by the control unit 120 of the image processing device 100,acquire images, and generate images. The generated images are input intothe integration processing unit 130.

(Input Device)

The input device 300 is a device for the user to input instructioninformation into the intraocular image processing system 1. The inputdevice 300 includes, for example, an input unit for the user to inputinformation including a mouse, a keyboard, a touch panel, a button, amicrophone, a switch, a lever, and the like, an input control circuitthat generates an input signal on the basis of the input by the user andoutputs the input signal to a CPU, and the like.

(Output Device)

The output device 400 is a device that displays the presentation image.The output device 400 may be, for example, a display device including aliquid crystal display (LCD) device, an organic light emitting diode(OLED) device, and the like. Furthermore, the output device 400 mayinclude a voice output device including a speaker and the like.

[3. Image Processing]

Next, with reference to FIG. 6, image integration processing andpresentation image generation processing by the intraocular imageprocessing system 1 will be described. FIG. 6 is a flowchart showing aflow of the image integration processing and the presentation imagegeneration processing by the intraocular image processing system 1according to the present embodiment. The processing shown in FIG. 6 isexecuted by the image processing device 100 of the intraocular imageprocessing system 1.

[3.1. Image Integration Processing]

To begin with, the image processing device 100 acquires images generatedby the image generating devices 200 (S10), and generates one integratedimage from the acquired images (S20).

Before detailed description of the image integration processing, anoutline of the image integration processing will be described withreference to FIGS. 7 and 8. FIG. 7 is an explanatory diagram showinggeneration processing of a three-dimensional integrated image. FIG. 8 isan explanatory diagram showing one example of a two-dimensionalintegrated image.

The integrated image is generated on the basis of the images acquiredfrom the eyeball by the image generating devices 200. For example, theimage generating devices 200 receive an instruction from the controlunit 120 and acquire intraocular information. With this operation, forexample, as shown in FIG. 7, surgical microscope images acquired by thesurgical microscope, tomographic acquisition device images acquired bythe tomographic acquisition device, endoscopic images acquired by theintraocular endoscope, or the like are generated. On the basis of theseimages and control information of the image generating devices 200 bythe control unit 120 as necessary, the integration processing unit 130integrates the images to generate an integrated image G.

The integrated image G may be a solid three-dimensional integrated imageas shown in FIG. 7, or may be a planar two-dimensional integrated imageas shown in FIG. 8. The integrated image G of FIG. 8 is an image inwhich an illustration g1 of the fundus generated by the illustrationgenerating device, a surgical microscope image g2, and an endoscopicimage g3 are integrated. Generation of such an integrated image obtainedby integrating the plurality of images into one image makes it possibleto simultaneously observe a wider range than an intraocular rangeindicated by the image generated by each image generating device. Theimage integration processing will be described in detail below.

(1) Image Acquisition

For example, it is assumed that a request for presenting the integratedimage from the user of the intraocular image processing system 1 isinput from the input device 300. On receipt of the presentation requestof the integrated image via the interface unit 110, the control unit 120of the image processing device 100 notifies the image generating devices200 of an image generation instruction, and causes the image generatingdevices 200 to output the generated images to the integration processingunit 130 (S10).

As the image generating devices 200, for example, a surgical microscope,a tomographic acquisition device, an intraocular endoscope, anillustration generating device, and the like are used.

(Image Acquisition by Surgical Microscope)

The surgical microscope is a microscope that observes the inside of theeye from a pupil and magnifies and presents a real image in anophthalmologic surgical operation. The surgical microscope includes animage capturing device for acquiring a microscope image. The range ofthe image acquired by the image capturing device included in thesurgical microscope differs depending on the optical system as shown inFIG. 1. For example, for posterior pole observation, an image of arelatively narrow range in the eye is acquired, and for wide-angleobservation, an image of a relatively wide range in the eye is acquired.

The image capturing device may include a right eye camera and a left eyecamera to enable images observed by the surgical microscope to bestereoscopically viewed. In this case, for example, as shown in FIG. 9,a right eye image g_(R) captured by the right eye camera and a left eyeimage g_(L) captured by the left eye camera are acquired. From aparallax obtained on the basis of the right eye image g_(R) and the lefteye image g_(L), for example, it is also possible to estimate a solidshape of the retina of the eyeball. Furthermore, on the basis of thesurgical microscope image, it is also possible to estimate the postureof the eye. For example, as shown in FIG. 10, it is possible to estimatethe posture of the eye with respect to the surgical microscope bystoring the position of the image feature including a corneal limbalblood vessel 31, a trocar 33, and the like in a situation where theeyeball is facing the surgical microscope when a surgical operationstarts and observing subsequent change in the position of the imagefeature.

(Image Acquisition by Tomographic Acquisition Device)

The tomographic acquisition device is a device that acquires a tomogramof the eyeball. For example, the OCT, an ultrasound device such as anultrasound biomicroscope (UBM), or the like may be used. Hereinafter, acase where the OCT is used as one example of the tomographic acquisitiondevice will be described.

The OCT can acquire a front image of the eyeball by acquiring a tomogramas volume data and generating a so-called enFace image on the basis ofsuch data. The enFace image is generated by taking an average, maximum,minimum, or the like of luminance values in a depth range set inadvance. As the depth range to be used to generate the enFace image, alldepths (D₀) may be used in an image g of a tomogram shown in FIG. 11,and a depth (D₁) constant regardless of a retina region may be used.Furthermore, in the image g of a tomogram, a depth in a range of aspecified region (layer) of the retina may be used, for example, a depthfrom an inner limiting membrane to an outer limiting membrane (D₂), adepth from the inner limiting membrane to an inner plexiform layer (D₃),and the like. From such an image of tomogram, for example, the frontimage of the eyeball shown in FIG. 12 is acquired as the image gacquired by the image generating device.

Furthermore, as the front image in the OCT, as shown in FIG. 13, forexample, it is possible to acquire one or more vascular plexus imagesthat can be acquired by OCT angiography. Such a vascular plexus imagemay be used as the image g acquired by the image generating device.

Note that the solid shape of the retina can also be estimated on thebasis of a depth position of the retina acquired from the image g of theOCT tomogram. For example, as shown in FIG. 14, a plurality of depths(for example, d₁, d₂, d₃) from an upper end of the image g to the innerlimiting membrane of the retina in a direction along a surface of theretina is acquired. The solid shape of the retina is estimated from thechange in the depth position of the retina.

Furthermore, the posture of the eye with respect to the OCT can also beestimated on the basis of the position of the specified region acquiredfrom the image g of the OCT tomogram. For example, as shown in FIG. 15,it is possible to measure the position of a corneal limbus 35 or aniridocorneal angle 37 and estimate the posture of the eye with respectto the OCT on the basis of a measurement result. Note that in a casewhere the OCT tomogram is acquired as volume data, the corneal limbus 35and the iridocorneal angle 37 are distributed in a ring shape.

Information regarding the estimated solid shape of the retina and theposture of the eye with respect to the OCT may be used in the generationprocessing of the integrated image.

(Image Acquisition by Intraocular Endoscope)

The intraocular endoscope is a surgical tool for observing the inside ofthe eyeball. The intraocular endoscope emits illumination light from thetip of an insertion portion that is inserted into the eye, illuminates aportion to be captured, and captures an image of the inside of the eye.The image captured by the intraocular endoscope can be integrated intothe integrated image.

When integrating the endoscopic image, a posture estimation result ofthe endoscope with respect to the eye may be used. Specifically, byestimating a range and direction of the image captured by theintraocular endoscope on the retina, on the basis of the estimationresult, it is possible to perform alignment with an image generated byanother image generating device. The posture of the intraocularendoscope with respect to the eye may be estimated, for example, on thebasis of the estimation result of the posture of the eye based on thesurgical microscope image and the estimation result of the posture ofthe intraocular endoscope (for example, Patent Document 1). At thistime, for example, as shown in FIG. 16, the intraocular endoscope 20 inwhich a three-dimensional marker 21 a for identifying the rotation angleor position is attached to the grip portion 21 is used and measurementis performed with an optical tracking system, thereby making it easierto acquire robust results. FIG. 17 shows one example of the endoscopicimage (image g) acquired by the intraocular endoscope 20, and the rangeB and direction of the endoscopic image with respect to the fundus rangeAo identified from the estimation result of the posture of theintraocular endoscope 20 when the endoscopic image is acquired. An arrowon the left side of FIG. 17 indicates the direction to which the upwarddirection of the endoscopic image on the right side of FIG. 17corresponds.

(Image Acquisition by Illustration Generating Device)

The surgical microscope, the tomographic acquisition device, and theintraocular endoscope described above are each a modality that canacquire the ophthalmologic image. An image other than the ophthalmologicimage may be used as an image to be integrated into the integratedimage. For example, the illustration generating device may be used togenerate an illustration that illustrates intraocular information, andthe illustration may be used as the image g to be integrated into theintegrated image. For example, as shown in FIG. 18, an illustrationrepresenting the fundus may be generated by the illustration generatingdevice. At this time, the illustration generating device may generatethe illustration, for example, on the basis of average fundusinformation obtained by dissection, or may generate the illustration onthe basis of information acquired about the eye of a patient.

Note that the illustration generated on the basis of the average fundusinformation obtained by dissection does not always agree with the fundusof the patient's eye. Therefore, in a case where such an illustration isintegrated into the integrated image, for example, geometrictransformation may be performed on the basis of the information acquiredabout the patient's eye so as to agree with the image acquired byanother image generating device.

In this way, respective images g acquired by the image generatingdevices 200 on the basis of the instruction from the control unit 120are output to the integration processing unit 130.

(2) Generation of Integrated Image

Next, the integration processing unit 130 generates one integrated imageG from the images g acquired by the image generating devices 200 (S20).The images g are integrated by correlating the intraocular positions.

To begin with, the integration processing unit 130 uses the image gacquired from one arbitrary image generating device 200 as a referenceimage. Then, the integration processing unit 130 aligns the image gacquired by another image generating device 200 with an image shape ofthe reference image, and integrates these images g, for example, afterfurther performing transformation processing and the like includingaffine transformation and the like. For example, in the integrated imageG shown in FIG. 8, the illustration g1 of the fundus generated by theillustration generating device is used as a reference image. Thesurgical microscope image g2 and the endoscopic image g3 are integratedon the basis of region positions of the eye, for example, the centralfovea, the ora serrata, and the like.

The images g to be integrated into the integrated image G are notlimited to the current ophthalmologic images, and for example, pastimages acquired by the image generating devices 200 may be used. In acase where images from a plurality of image generating devices andfurthermore past images are integrated, a plurality of images at thesame position in the eye may exist. In this case, one image may begenerated by performing weighted addition of images according toidentity of the images, and only one image may be selected and used fromamong the plurality of images. For example, in a case where weightedaddition of images is performed on a past image, the weight of the imagemay be reduced, and in a case where only one image is selected fromamong a plurality of images, the priority of selection may be lowered.It is expected that a good image will be obtained for the user byperforming such processing.

Furthermore, for example, in a case where the solid integrated image Gas shown in FIG. 7 is generated, for example, the posture of the eye,the depth of the retina, and the posture of the intraocular endoscopemay be estimated from the images g, and the images g may be integratedon the basis of estimation results. Alternatively, the integrationprocessing unit 130 may recognize the image feature of each image g andestimate the positional relationship of the plurality of images fromposition distribution of the recognized image feature. The positions ofthe plurality of images g can also be aligned on the basis of such anestimation result. As the image feature, for example, macro informationsuch as positions of regions including the optic disc, central fovea,ora serrata, and the like may be used, or micro information such as abranch point of a blood vessel may be used.

Furthermore, in a case where the integrated image G is built into asolid retinal image, as a shape of the retina, for example, a perfectsphere model may be used, an average retina shape model may be used, anda solid model of the patient's eye restored on the basis of the solidinformation obtained from the images g may be used. Note that toestimate distortion of a front image generated by each image generatingdevice 200 and to estimate the shape of the retina, a model regardingoptical characteristics of the intermediate translucent body includingthe optical system used for each image generating device 200 and thelike may be used.

In this way, the images g acquired by respective image generatingdevices 200 are integrated to generate one integrated image G. Theintegrated image G shows a wider range than a range of at least anophthalmologic image acquired by a modality including the surgicalmicroscope, the tomographic acquisition device, the intraocularendoscope, and the like out of respective image generating devices 200.Such an integrated image G makes it possible to simultaneously presentwide-range intraocular information that is not obtained with only onemodality.

[3.2. Presentation Image Generation Processing]

Returning to the description of FIG. 6, when the integrated image G isgenerated by the integration processing unit 130, the presentation imageis generated by the presentation processing unit 140 (S30). Thereafter,the presentation image is presented to the user (S40). The presentationprocessing unit 140 works the integrated image G to generate thepresentation image, thereby making it possible to present to the user agood image that is comfortable and easy to recognize. The presentationprocessing unit 140 may perform, for example, the following work on theintegrated image G.

(Flattening Solid Integrated Image)

In a case where a solid integrated image is generated as the integratedimage G, in order to make the presentation image a planar image, it isrequired at least to project the integrated image G onto a planar imageto make the image two-dimensional. As a method of projecting the solidintegrated image G, for example, projection methods shown in FIGS. 19 to21 are considered.

For example, as shown in FIG. 19, the solid integrated image G may beorthographically projected on a projection plane S to make a planarimage. To orthographically project the integrated image G, it isnecessary to determine an angle of the projection plane S with respectto the eye E. Furthermore, for example, as shown in FIG. 20, the solidintegrated image G may be perspectively projected on the projectionplane S to make a planar image. To perspectively project the integratedimage G, it is necessary to determine positions of a viewpoint P and theprojection plane S with respect to the eye E. Alternatively, forexample, as a method of projecting the solid integrated image G, theintegrated image G may be cylindrically projected to make a planarimage. That is, the integrated image G is represented by the Mercatorprojection. To change the integrated image G to a planar image by usingthe Mercator projection, as shown in FIG. 21, it is necessary todetermine a position of a central axis C with respect to the eye E and aline Lc corresponding to 0 degrees longitude.

For the projection plane S, viewpoint P, central axis C, and line Lc of0 degrees longitude, setting information set in advance on the basis ofa mode designated by the user and the like may be used. For example, theprojection plane S, viewpoint P, central axis C, and line Lc of 0degrees longitude may be variable according to the posture of the eyeand the position of the surgical tool. For example, if the direction ofthe optical axis of the surgical microscope with respect to the eye isused as a normal line to the projection plane S, an image that issensuously close to the image obtained by the surgical microscope isobtained. Such a setting is expected to reduce the user's discomfortwith the presentation image. Furthermore, by setting a perpendicularline from the surgical tool to the retina as a normal line to theprojection plane S, or by setting the vicinity of the surgical tool asthe position of the viewpoint P, it is assumed that possibility ofacquiring an image regarding the user's attention target will beincreased

(Geometric Transformation Processing)

Conventionally, it has been difficult to enlarge and observe theposterior pole while observing the fundus at a wide angle because theimages cannot be acquired simultaneously. In contrast, the intraocularimage processing system 1 according to the present embodiment canpresent a wide angle range and an enlarged view of the posterior polesimultaneously by performing image transformation on the integratedimage G.

FIG. 22 shows one example of geometric transformation of the integratedimage G. In such geometric transformation, the integrated image G istransformed by changing a distance on the basis of a function set inadvance without changing a direction of a pixel position from areference point. The function can be set as needed, for example, asshown on the left side of FIG. 22, a function that performstransformation may be used such that a distance ratio beforetransformation is different from the distance ratio after transformationaccording to a predetermined distance from the origin. Note that thereference point may be, for example, fixed to the origin O that is thecenter of the integrated image G, or may be set on the basis of aspecified region, for example, the position of the macula or optic disc,the position of the surgical tool, or the like. By transforming theintegrated image G by using such a function, the posterior pole 56 nearthe origin O, which is the reference point, is uniformly enlarged, andthe other portions are reduced. As a result, the integrated image G istransformed into an image H that allows good observation of both theentire fundus and the posterior pole.

Furthermore, if a function other than the function shown on the leftside of FIG. 22 is used, another geometric transformation is performedon the integrated image G. For example, if the integrated image G istransformed using the function shown in the upper right of FIG. 23, animage Ha having pincushion distortion can be obtained. For example, in acase where the integrated image G before transformation is an imageacquired using a wide-angle observation lens and depicts the retina in aplanar fashion, such a transformation brings the retina closer to aspherical shape, making it easier to understand intraocular informationfrom the image.

In addition, for example, if the function as shown in the lower right ofFIG. 23 is used, the integrated image G can be transformed into an imageHb having barrel distortion. Furthermore, for example, if the functionshown in the upper right of FIG. 24 is used, the integrated image G canbe transformed into an image Hc having so-called Jingasa type distortionin which both pincushion distortion and barrel distortion appear, withthe central portion bulging outward and curves expanding again in theperipheral portion. Moreover, for example, if the function shown in thelower right of FIG. 24 is used, the integrated image G can betransformed into an image Hd having distortion in which both pincushiondistortion and barrel distortion appear, with the central portionshrinking inward and the peripheral portion bulging. These functions canbe appropriately selected according to preference of the user or thepurpose of observation.

(Image Work Indicating Type of Original Image)

The presentation processing unit 140 may work the images g so as toindicate the type of original images g when generating a presentationimage H from the integrated image G. As the type of original images, forexample, a type according to acquisition time of the images may beshown. That is, the images g are worked to allow distinction between apast image and a current image. As a method of work, for example, thecurrent image may be displayed as it is, and the past image may undergocolor transformation by using only grayscale or specified color channel.Furthermore, in a case where the type of original image g is shownaccording to the type of image generating device 200, for example, theimage may be worked by adjusting the luminosity or sharpness of theacquired image g.

As one example of the presentation image H, FIG. 25 shows thepresentation image H in which for the integrated image of anillustration and an endoscopic image, past images out of the endoscopicimage are changed to a grayscale. In the presentation image H shown inFIG. 25, the illustration g1 and the current endoscopic image g2 areimages identical to images when integrated. Meanwhile, the pastendoscopic images h are images changed to a grayscale by thepresentation processing unit 140. In this way, by indicating the type oforiginal image, it becomes possible to present more information with thepresentation image H.

(Integrated Image and Single Image Arranged Side by Side)

Depending on details of technique and the type of image generatingdevice 200, it may be more preferable for the user, such as easy toperform technique, to present the image itself generated by the imagegenerating device 200 together with the integrated image than to presentonly the integrated image generated as described above. In this case,the image generated by the image generating device 200 may be presentedtogether with the integrated image side by side.

FIGS. 26 and 27 show examples in which the illustration and theendoscopic image captured by the intraocular endoscope are presentedside by side. Both FIGS. 26 and 27 show the integrated image G of theillustration and the endoscopic image on the left side, and only theendoscopic image (image g) on the right side. Note that in order toreduce the processing load of generating the integrated image, as theendoscopic image to be integrated with the illustration, a markindicating the range of the endoscopic image is used instead of theendoscopic image itself. Therefore, the integrated image G is generatedby integrating the illustration and the mark indicating the range of theendoscopic image (range B).

When arranging the integrated image and the endoscopic image side byside, for example, as shown in FIG. 26, the presentation direction ofthe endoscopic image may be aligned with the direction of the image inthe integrated image. Alternatively, as shown in FIG. 27, the endoscopicimage may be presented such that the upward direction of the endoscopicimage is vertically upward, and in the integrated image G, a symbol (forexample, an arrow) indicating the direction corresponding to the upwarddirection of the endoscopic image may be presented together with therange B of the endoscopic image. By arranging the integrated image andthe image generated by the image generating device 200 side by side inthis manner, the correspondence between the two images can be presentedto the user in an easy-to-understand manner. Note that in FIGS. 26 and27, only one image g is arranged side by side with the integrated image,but the present disclosure is not limited to such an example. Aplurality of images g may be presented side by side with the integratedimage.

[4. Hardware Configuration]

A hardware configuration example of the image processing device 100 ofthe intraocular image processing system 1 according to the aboveembodiment will be described. FIG. 28 is a hardware configurationdiagram showing the hardware configuration of the image processingdevice 100 according to the above embodiment.

The image processing device 100 according to the present embodiment canbe implemented by a processing device including a computer and the likeas described above. As shown in FIG. 28, the image processing device 100includes a central processing unit (CPU) 901, a read only memory (ROM)902, a random access memory (RAM) 903, and a host bus 904 a.Furthermore, the image processing device 100 includes a bridge 904, anexternal bus 904 b, an interface 905, an input device 906, an outputdevice 907, a storage device 908, a drive 909, a connection port 911,and a communication device 913.

The CPU 901 functions as an arithmetic processing device and a controldevice, and controls the overall operation in the image processingdevice 100 in accordance with various programs. Furthermore, the CPU 901may be a microprocessor. The ROM 902 stores programs, calculationparameters, and the like used by the CPU 901. The RAM 903 temporarilystores programs used in the execution of the CPU 901, parameters thatappropriately change in the execution, and the like. These are connectedto each other by the host bus 904 a including a CPU bus and the like.

The host bus 904 a is connected to the external bus 904 b such as aperipheral component interconnect/interface (PCI) bus via the bridge904. Note that the host bus 904 a, the bridge 904, and the external bus904 b do not necessarily need to be separated, and these functions maybe implemented in one bus.

The input device 906 includes an input unit for the user to inputinformation such as a mouse, a keyboard, a touch panel, a button, amicrophone, a switch, and a lever, an input control circuit thatgenerates an input signal on the basis of an input by the user andoutputs the input signal to the CPU 901, and the like. The output device907 includes, for example, a display device such as a liquid crystaldisplay (LCD) device, an organic light emitting diode (OLED) device, anda lamp, and a voice output device such as a speaker.

The storage device 908 is one example of a storage unit of the imageprocessing device 100, and is a device for data storage. The storagedevice 908 may include a storage medium, a recording device that recordsdata in the storage medium, a reading device that reads data from thestorage medium, an erasing device that erases data recorded in thestorage medium, and the like. This storage device 908 drives a hard diskand stores programs executed by the CPU 901 or various data.

The drive 909 is a reader-writer for a storage medium, and is built inor externally attached to the image processing device 100. The drive 909reads information recorded on a mounted removable recording medium suchas a magnetic disk, optical disk, magneto-optical disk, semiconductormemory, or the like, and outputs the information to the RAM 903.

The connection port 911 is an interface to be connected to an externaldevice, and is a connection port to an external device capable oftransmitting data by, for example, universal serial bus (USB) and thelike. Furthermore, the communication device 913 is a communicationinterface including, for example, a communication device for connectingto a communication network 5 and the like. Furthermore, thecommunication device 913 may be a wireless local area network(LAN)-compatible communication device, a wireless USB-compatiblecommunication device, or a wire communication device that performs wiredcommunication.

[5. Conclusion]

The configuration of the intraocular image processing system 1 accordingto the present disclosure and the image processing based on theconfiguration have been described above. Such an intraocular imageprocessing system 1 integrates images generated by the plurality ofimage generating devices 200 to generate one integrated image G. Theintegrated image G shows a wider range than a range of at least anophthalmologic image acquired by a modality including the surgicalmicroscope, the tomographic acquisition device, the intraocularendoscope, and the like out of respective image generating devices 200.Such an integrated image G makes it possible to simultaneously presentwide-range intraocular information that is not obtained with only onemodality.

The integrated image G may be a planar image, or can be constructed as asolid model including a sphere model and the like. Generation of thesolid integrated image G makes it possible to present a presentationimage that allows easy recognition that the retina is spherical.

Furthermore, in the presentation of the integrated image G, for example,by adding work to the integrated image G, information to be presentedsimultaneously in the integrated image G can be presented to the user tofacilitate recognition more easily. For example, it becomes possible topresent a wide range of retina and an enlarged portion of the specifiedportion simultaneously. Furthermore, the geometric transformation of theintegrated image G makes it possible to eliminate restrictions due tooptical characteristics of the intermediate translucent body to theretina including the optical system of the image generating device 200,and to present an image having a shape that is easy for the user to lookat.

Moreover, the presentation method may be changed according to the typeof image before being integrated. For example, when presenting aplurality of images acquired by the same image generating device 200 atdifferent times, the images may be worked according to the acquisitiontime. With this configuration, the use of a past image as the image tobe integrated makes it possible to also recognize that the past image isnot a current image while extending the range of images to besimultaneously obtained by the integrated image G. Furthermore, theintegrated image G and the image itself generated by the imagegenerating device 200 may be presented side by side. For example, forthe endoscopic image, the user can more easily understand the endoscopicimage in some cases by presenting the endoscopic image with theintegrated image G side by side than presenting the endoscopic image ina state of being integrated in the integrated image G. In this way, theintegrated image G and the image itself generated by the imagegenerating device 200 may be arranged side by side according to detailsof technique and the type of image generating device 200.

The preferred embodiment of the present disclosure has been described indetail above with reference to the accompanying drawings, but thetechnical scope of the present disclosure is not limited to such anexample. It is obvious that persons of ordinary skill in the technicalfield of the present disclosure can conceive various modifications oralterations within the scope of the technical idea described in theclaims, and it is of course understood that these also fall within thetechnical scope of the present disclosure.

Furthermore, effects described in the present specification are merelydescriptive or illustrative and not restrictive. That is, the technologyaccording to the present disclosure can produce other effects obvious tothose skilled in the art from the description in the presentspecification, in addition to or instead of the effects described above.

Note that the following configurations also belong to the technicalscope of the present disclosure.

(1)

An image processing device including an integration processing unitconfigured to integrate at least a plurality of intraoperativelyacquired images acquired during an ophthalmologic surgical operation bycorrelating intraocular positions and to generate one integrated imageindicating intraocular information in a range wider than an image rangeof each of the intraoperatively acquired images.

(2)

The image processing device according to (1) described above, in whichthe intraoperatively acquired images include an image acquired by usinga surgical microscope.

(3)

The image processing device according to (1) or (2) described above, inwhich the intraoperatively acquired images include an image acquired byusing a tomographic acquisition device.

(4)

The image processing device according to any one of (1) to (3) describedabove, in which the intraoperatively acquired images include an imageacquired by using an intraocular endoscope.

(5)

The image processing device according to any one of (1) to (4) describedabove, in which the integration processing unit integrates anillustration regarding a fundus with the intraoperatively acquiredimages to generate the integrated image.

(6)

The image processing device according to any one of (1) to (5) describedabove, in which the integrated image is a solid image.

(7)

The image processing device according to any one of (1) to (6) describedabove, in which the integration processing unit models the integratedimage into a substantially spherical shape.

(8)

The image processing device according to any one of (1) to (6) describedabove, in which the integration processing unit models the integratedimage into a solid shape acquired on the basis of the intraoperativelyacquired images.

(9)

The image processing device according to any one of (1) to (8) describedabove, in which the integration processing unit identifies theintraocular positions of the intraoperatively acquired images on thebasis of information included in the intraoperatively acquired imagesand integrates the intraoperatively acquired images.

(10)

The image processing device according to any one of (1) to (9) describedabove, further including a presentation processing unit configured toprocess the integrated image according to a presentation format of theintegrated image.

(11)

The image processing device according to (10) described above, in whichthe presentation processing unit transforms the integrated image that isa solid image into a planar image.

(12)

The image processing device according to (10) described above, in whichthe presentation processing unit performs geometric transformation onthe integrated image.

(13)

The image processing device according to any one of (10) to (12)described above, in which the presentation processing unit presents theintegrated image to allow identification of types of theintraoperatively acquired images integrated into the integrated image.

(14)

The image processing device according to any one of (10) to (13)described above, in which the presentation processing unit presents theintegrated image and at least one of the intraoperatively acquiredimages arranged side by side.

(15)

An image processing method including: integrating at least a pluralityof intraoperatively acquired images acquired during an ophthalmologicsurgical operation by correlating intraocular positions; and generatingone integrated image indicating intraocular information in a range widerthan an image range of each of the intraoperatively acquired images.

(16)

An intraocular image processing system including:

a plurality of image generating devices configured to acquireintraoperatively acquired images during an ophthalmologic surgicaloperation; and

an image processing device configured to integrate at least theplurality of intraoperatively acquired images acquired by the imagegenerating devices by correlating intraocular positions and to generateone integrated image indicating intraocular information in a range widerthan an image range of each of the intraoperatively acquired images.

(17)

An image processing device including an integration processing unitconfigured to integrate ophthalmologic images acquired by a plurality oftypes of ophthalmologic observation devices by correlating intraocularpositions and to generate one integrated image indicating intraocularinformation in a range wider than a range indicated by each of theophthalmologic images.

(18)

The image processing device according to (17) described above, in whichthe ophthalmologic images are images acquired intraoperatively.

(19)

The image processing device according to (17) or (18) described above,in which the ophthalmologic images include an image acquired by using asurgical microscope.

(20)

The image processing device according to any one of (17) to (19)described above, in which the ophthalmologic images include an imageacquired by using a tomographic acquisition device.

(21)

The image processing device according to any one of (17) to (20)described above, in which the ophthalmologic images include an imageacquired by using an intraocular endoscope.

(22)

The image processing device according to any one of (17) to (21)described above, in which the integration processing unit integrates anillustration regarding a fundus with the ophthalmologic images togenerate the integrated image.

(23)

The image processing device according to any one of (17) to (22)described above, in which the integrated image is a solid image.

(24)

The image processing device according to any one of (17) to (22)described above, in which the integration processing unit models theintegrated image into a substantially spherical shape.

(25)

The image processing device according to any one of (17) to (24)described above, in which the integration processing unit models theintegrated image into a solid shape acquired on the basis of theophthalmologic images.

(26)

The image processing device according to any one of (17) to (25)described above, in which the integration processing unit identifiesintraocular positions of the ophthalmologic images on the basis ofinformation included in the ophthalmologic images and integrates theophthalmologic images.

(27)

The image processing device according to any one of (17) to (26)described above, further including a presentation processing unitconfigured to process the integrated image according to a presentationformat of the integrated image.

(28)

The image processing device according to (27) described above, in whichthe presentation processing unit transforms the integrated image that isa solid image into a planar image.

(29)

The image processing device according to (27) described above, in whichthe presentation processing unit performs geometric transformation onthe integrated image.

(30)

The image processing device according to any one of (27) to (29)described above, in which the presentation processing unit presents theintegrated image to allow identification of types of the ophthalmologicimages integrated into the integrated image.

(31)

The image processing device according to any one of (27) to (30)described above, in which the presentation processing unit presents theintegrated image and at least one of the ophthalmologic images arrangedside by side.

(32)

An image processing method including: integrating ophthalmologic imagesacquired by a plurality of types of ophthalmologic observation devicesby correlating intraocular positions; and generating one integratedimage indicating intraocular information in a range wider than a rangeindicated by each of the ophthalmologic images.

(33)

An intraocular image processing system including:

a plurality of ophthalmologic observation devices configured to acquireophthalmologic images during an ophthalmologic surgical operation; and

an image processing device configured to integrate a plurality of typesof the ophthalmologic images acquired by the ophthalmologic observationdevices by correlating intraocular positions and to generate oneintegrated image indicating intraocular information in a range widerthan a range indicated by each of the ophthalmologic images.

REFERENCE SIGNS LIST

-   1 Intraocular image processing system-   10 a Contact lens-   10 b Wide-angle observation lens-   20 Intraocular endoscope-   21 Grip portion-   21 a Three-dimensional marker-   23 Tip-   31 Corneal limbal blood vessel-   33 Trocar-   35 Corneal limbus-   37 Iridocorneal angle-   53 Crystalline lens-   55 Vitreous body-   56 Posterior pole-   57 Retina-   58 Ora serrata-   59 Pars plana-   100 Image processing device-   110 Interface unit-   120 Control unit-   130 Integration processing unit-   140 Presentation processing unit-   200 Image generating device-   300 Input device-   400 Output device

1. An image processing device comprising an integration processing unitconfigured to integrate ophthalmologic images acquired by a plurality oftypes of ophthalmologic observation devices by correlating intraocularpositions and to generate one integrated image indicating intraocularinformation in a range wider than a range indicated by each of theophthalmologic images.
 2. The image processing device according to claim1, wherein the ophthalmologic images are images acquiredintraoperatively.
 3. The image processing device according to claim 1,wherein the ophthalmologic images include an image acquired by using asurgical microscope.
 4. The image processing device according to claim1, wherein the ophthalmologic images include an image acquired by usinga tomographic acquisition device.
 5. The image processing deviceaccording to claim 1, wherein the ophthalmologic images include an imageacquired by using an intraocular endoscope.
 6. The image processingdevice according to claim 1, wherein the integration processing unitintegrates an illustration regarding a fundus with the ophthalmologicimages to generate the integrated image.
 7. The image processing deviceaccording to claim 1, wherein the integrated image is a solid image. 8.The image processing device according to claim 1, wherein theintegration processing unit models the integrated image into asubstantially spherical shape.
 9. The image processing device accordingto claim 1, wherein the integration processing unit models theintegrated image into a solid shape acquired on a basis of theophthalmologic images.
 10. The image processing device according toclaim 1, wherein the integration processing unit identifies intraocularpositions of the ophthalmologic images on a basis of informationincluded in the ophthalmologic images and integrates the ophthalmologicimages.
 11. The image processing device according to claim 1, furthercomprising a presentation processing unit configured to process theintegrated image according to a presentation format of the integratedimage.
 12. The image processing device according to claim 11, whereinthe presentation processing unit transforms the integrated image that isa solid image into a planar image.
 13. The image processing deviceaccording to claim 11, wherein the presentation processing unit performsgeometric transformation on the integrated image.
 14. The imageprocessing device according to claim 11, wherein the presentationprocessing unit presents the integrated image to allow identification oftypes of the ophthalmologic images integrated into the integrated image.15. The image processing device according to claim 11, wherein thepresentation processing unit presents the integrated image and at leastone of the ophthalmologic images arranged side by side.
 16. An imageprocessing method comprising: integrating ophthalmologic images acquiredby a plurality of types of ophthalmologic observation devices bycorrelating intraocular positions; and generating one integrated imageindicating intraocular information in a range wider than a rangeindicated by each of the ophthalmologic images.
 17. An intraocular imageprocessing system comprising: a plurality of ophthalmologic observationdevices configured to acquire ophthalmologic images during anophthalmologic surgical operation; and an image processing deviceconfigured to integrate a plurality of types of the ophthalmologicimages acquired by the ophthalmologic observation devices by correlatingintraocular positions and to generate one integrated image indicatingintraocular information in a range wider than a range indicated by eachof the ophthalmologic images.